Module testing adjustment and configuration

ABSTRACT

In one embodiment, a method for testing adjustment and configuration is disclosed. The method can include accessing source code of a test framework that is configured for testing a module, creating a configuration folder having a property override for a test suite for the module testing, determining a source root folder for the test suite, starting the test framework by passing in an identifier for the test suite, and adding a custom test to the source root folder using the configuration folder to customize the test suite. The method can further include compiling the test framework with each of the plurality of test folders enabled. The method also may use a refactoring tool to make changes in a file within the test framework.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF THE INVENTION

The current invention relates generally to adjusting a software testingframework in a multi-tenant database network system.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches, which in and of themselves may also be inventions.

In conventional database systems, users access their data resources inone logical database. A user of such a conventional system typicallyretrieves data from and stores data on the system using the user's ownsystems. A user system might remotely access one of a plurality ofserver systems that might in turn access the database system. Dataretrieval from the system might include the issuance of a query from theuser system to the database system. The database system might processthe request for information received in the query and send to the usersystem information relevant to the request. Unfortunately, it isdifficult in many conventional approaches to adjust and customize testframeworks to suit the needs of a particular application.

BRIEF SUMMARY

In accordance with embodiments, there are provided mechanisms andmethods for testing modules. Such approaches can enable access of sourcecode of a test framework that is configured for testing a module in themulti-tenant database system. The approach further includes creating aconfiguration folder having a property override for a test suite for themodule testing and determining a source root folder for the test suite.These approaches can enable starting the test framework by passing in anidentifier for the test suite and adding a custom test to the sourceroot folder using the configuration folder to customize the test suite.

In an embodiment and by way of example, a method for testing a module isprovided. This particular example method includes accessing source codeof a test framework that is configured for testing a module in themulti-tenant database system, creating a configuration folder having aproperty override for a test suite for the module testing, determining asource root folder for the test suite, starting the test framework bypassing in an identifier for the test suite, and adding a custom test tothe source root folder using the configuration folder to customize thetest suite.

While the present invention is described with reference to an embodimentin which techniques for testing in a multi-tenant database environmentare implemented in a system having an application server providing afront end for an on-demand database service capable of supportingmultiple tenants, the present invention is not limited to multi-tenantdatabases nor deployment on application servers. Embodiments may bepracticed using other database architectures, i.e., ORACLE®, DB2® by IBMand the like without departing from the scope of the embodimentsclaimed.

Any of the above embodiments may be used alone or together with oneanother in any combination. Inventions encompassed within thisspecification may also include embodiments that are only partiallymentioned or alluded to or are not mentioned or alluded to at all inthis brief summary or in the abstract. Although various embodiments ofthe invention may have been motivated by various deficiencies with theprior art, which may be discussed or alluded to in one or more places inthe specification, the embodiments of the invention do not necessarilyaddress any of these deficiencies. In other words, different embodimentsof the invention may address different deficiencies that may bediscussed in the specification. Some embodiments may only partiallyaddress some deficiencies or just one deficiency that may be discussedin the specification, and some embodiments may not address any of thesedeficiencies.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following drawings like reference numbers are used to refer tolike elements. Although the following figures depict various examples ofthe invention, the invention is not limited to the examples depicted inthe figures.

FIG. 1 illustrates a block diagram of an example of an environment wherean on-demand database service might be used;

FIG. 2 illustrates a block diagram of an embodiment of elements of FIG.1 and various possible interconnections between these elements;

FIG. 3 illustrates an example simplified flow diagram for testing amodule; and

FIG. 4 illustrates a more detailed flow diagram of the simplified flowdiagram exemplified in FIG. 3.

DETAILED DESCRIPTION

General Overview

Systems and methods are provided for testing modules, and adjusting andconfiguration of test frameworks.

As used herein, the term multi-tenant database system refers to thosesystems in which various elements of hardware and software of thedatabase system may be shared by one or more customers. For example, agiven application server may simultaneously process requests for a greatnumber of customers, and a given database table may store rows for apotentially much greater number of customers. As used herein, the termquery plan refers to a set of steps used to access information in adatabase system.

Next, mechanisms and methods for testing modules will be described withreference to example embodiments.

System Overview

FIG. 1 illustrates a block diagram of an environment 110 where anon-demand database service might be used. Environment 110 may includeuser systems 112, network 114, system 116, processor system 117,application platform 118, network interface 120, tenant data storage122, system data storage 124, program code 126, and process space 128.In other embodiments, environment 110 may not have all of the componentslisted and/or may have other elements instead of, or in addition to,those listed above.

Environment 110 is an environment in which an on-demand database serviceexists. User system 112 may be any machine or system that is used by auser to access a database user system 116. For example, any of usersystems 112 can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Asillustrated in FIG. 1 (and in more detail in FIG. 2) user systems 112might interact via a network 114 with an on-demand database service,which is system 116.

An on-demand database service, such as system 116, is a database systemthat is made available to outside users that do not need to necessarilybe concerned with building and/or maintaining the database system, butinstead may be available for their use when the users need the databasesystem (e.g., on the demand of the users). Some on-demand databaseservices may store information from one or more tenants stored intotables of a common database image to form a multi-tenant database system(MTS). Accordingly, “on-demand database service 116” and “system 116”will be used interchangeably herein. A database image may include one ormore database objects. A relational database management system (RDMS) orthe equivalent may execute storage and retrieval of information againstthe database object(s). Application platform 118 may be a framework thatallows the applications of system 116 to run, such as the hardwareand/or software, e.g., the operating system. In an embodiment, on-demanddatabase service 116 may include an application platform 118 thatenables creating, managing and executing one or more applicationsdeveloped by the provider of the on-demand database service, usersaccessing the on-demand database service via user systems 112, or thirdparty application developers accessing the on-demand database servicevia user systems 112.

The users of user systems 112 may differ in their respective capacities,and the capacity of a particular user system 112 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 112 tointeract with system 116, that user system 112 has the capacitiesallotted to that salesperson. However, while an administrator is usingthat user system 112 to interact with system 116, that user system hasthe capacities allotted to that administrator. In systems with ahierarchical role model, users at one permission level may have accessto applications, data, and database information accessible by a lowerpermission level user, but may not have access to certain applications,database information, and data accessible by a user at a higherpermission level. Thus, different users will have different capabilitieswith regard to accessing and modifying application and databaseinformation, depending on a user's security or permission level.

Network 114 is any network or combination of networks of devices thatcommunicate with one another. For example, network 114 can be any one orany combination of a LAN (local area network), WAN (wide area network),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a TCP/IP (Transfer Control Protocol and Internet Protocol)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I,” that network will be used in manyof the examples herein. However, it should be understood that thenetworks that the present invention might use are not so limited,although TCP/IP is a frequently implemented protocol.

User systems 112 might communicate with system 116 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTPis used, user system 112 might include an HTTP client commonly referredto as a “browser” for sending and receiving HTTP messages to and from anHTTP server at system 116. Such an HTTP server might be implemented asthe sole network interface between system 116 and network 114, but othertechniques might be used as well or instead. In some implementations,the interface between system 116 and network 114 includes load sharingfunctionality, such as round-robin HTTP request distributors to balanceloads and distribute incoming HTTP requests evenly over a plurality ofservers. At least as for the users that are accessing that server, eachof the plurality of servers has access to the MTS' data; however, otheralternative configurations may be used instead.

In one embodiment, system 116, shown in FIG. 1, implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 116 includes application servers configured toimplement and execute CRM software applications as well as providerelated data, code, forms, webpages and other information to and fromuser systems 112 and to store to, and retrieve from, a database systemrelated data, objects, and Webpage content. With a multi-tenant system,data for multiple tenants may be stored in the same physical databaseobject, however, tenant data typically is arranged so that data of onetenant is kept logically separate from that of other tenants so that onetenant does not have access to another tenant's data, unless such datais expressly shared. In certain embodiments, system 116 implementsapplications other than, or in addition to, a CRM application. Forexample, system 16 may provide tenant access to multiple hosted(standard and custom) applications, including a CRM application. User(or third party developer) applications, which may or may not includeCRM, may be supported by application platform 118, which managescreation, storage of the applications into one or more database objectsand executing of the applications in a virtual machine in the processspace of system 116.

One arrangement for elements of system 116 is shown in FIG. 1, includinga network interface 120, application platform 118, tenant data storage122 for tenant data 123, system data storage 124 for system data 125accessible to system 116 and possibly multiple tenants, program code 126for implementing various functions of system 116, and a process space128 for executing MTS system processes and tenant-specific processes,such as running applications as part of an application hosting service.Additional processes that may execute on system 116 include databaseindexing processes.

Several elements in the system shown in FIG. 1 include conventional,well-known elements that are explained only briefly here. For example,each user system 112 could include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 112 typically runs an HTTP client, e.g., abrowsing program, such as Microsoft's Internet Explorer browser,Netscape's Navigator browser, Opera's browser, or a WAP-enabled browserin the case of a cell phone, PDA or other wireless device, or the like,allowing a user (e.g., subscriber of the multi-tenant database system)of user system 112 to access, process and view information, pages andapplications available to it from system 116 over network 114. Each usersystem 112 also typically includes one or more user interface devices,such as a keyboard, a mouse, trackball, touch pad, touch screen, pen orthe like, for interacting with a graphical user interface (GUI) providedby the browser on a display (e.g., a monitor screen, LCD display, etc.)in conjunction with pages, forms, applications and other informationprovided by system 116 or other systems or servers. For example, theuser interface device can be used to access data and applications hostedby system 116, and to perform searches on stored data, and otherwiseallow a user to interact with various GUI pages that may be presented toa user. As discussed above, embodiments are suitable for use with theInternet, which refers to a specific global internetwork of networks.However, it should be understood that other networks can be used insteadof the Internet, such as an intranet, an extranet, a virtual privatenetwork (VPN), a non-TCP/IP based network, any LAN or WAN or the like.

According to one embodiment, each user system 112 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Pentium® processor or the like. Similarly, system 116(and additional instances of an MTS, where more than one is present) andall of their components might be operator configurable usingapplication(s) including computer code to run using a central processingunit such as processor system 117, which may include an Intel Pentium®processor or the like, and/or multiple processor units. A computerprogram product embodiment includes a machine-readable storage medium(media) having instructions stored thereon/in which can be used toprogram a computer to perform any of the processes of the embodimentsdescribed herein. Computer code for operating and configuring system 116to intercommunicate and to process webpages, applications and other dataand media content as described herein are preferably downloaded andstored on a hard disk, but the entire program code, or portions thereof,may also be stored in any other volatile or non-volatile memory mediumor device as is well known, such as a ROM or RAM, or provided on anymedia capable of storing program code, such as any type of rotatingmedia including floppy disks, optical disks, digital versatile disk(DVD), compact disk (CD), microdrive, and magneto-optical disks, andmagnetic or optical cards, nanosystems (including molecular memory ICs),or any type of media or device suitable for storing instructions and/ordata. Additionally, the entire program code, or portions thereof, may betransmitted and downloaded from a software source over a transmissionmedium, e.g., over the Internet, or from another server, as is wellknown, or transmitted over any other conventional network connection asis well known (e.g., extranet, VPN, LAN, etc.) using any communicationmedium and protocols (e.g., TCP/IP, HTTP, HTTPS, Ethernet, etc.) as arewell known. It will also be appreciated that computer code forimplementing embodiments of the present invention can be implemented inany programming language that can be executed on a client system and/orserver or server system such as, for example, C, C++, HTML, any othermarkup language, Java™, JavaScript, ActiveX, any other scriptinglanguage, such as VBScript, and many other programming languages as arewell known may be used. (Java™ is a trademark of Sun Microsystems,Inc.).

According to one embodiment, each system 116 is configured to providewebpages, forms, applications, data and media content to user (client)systems 112 to support the access by user systems 112 as tenants ofsystem 116. As such, system 116 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB). As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., OODBMS or RDBMS) as is well known in the art. It should also beunderstood that “server system” and “server” are often usedinterchangeably herein. Similarly, the database object described hereincan be implemented as single databases, a distributed database, acollection of distributed databases, a database with redundant online oroffline backups or other redundancies, etc., and might include adistributed database or storage network and associated processingintelligence.

FIG. 2 also illustrates environment 110. However, in FIG. 2 elements ofsystem 116 and various interconnections in an embodiment are furtherillustrated. FIG. 2 shows that user system 112 may include processorsystem 112A, memory system 112B, input system 112C, and output system112D. FIG. 2 shows network 114 and system 116. FIG. 2 also shows thatsystem 116 may include tenant data storage 122, tenant data 123, systemdata storage 124, system data 125, User Interface (UI) 230, ApplicationProgram Interface (API) 232, PL/SOQL 234, save routines 236, applicationsetup mechanism 238, applications servers 200 ₁-200 _(N), system processspace 202, tenant process spaces 204, tenant management process space210, tenant storage area 212, user storage 214, and application metadata216. In other embodiments, environment 110 may not have the sameelements as those listed above and/or may have other elements insteadof, or in addition to, those listed above.

User system 112, network 114, system 116, tenant data storage 122, andsystem data storage 124 were discussed above in FIG. 1. Regarding usersystem 112, processor system 112A may be any combination of one or moreprocessors. Memory system 112B may be any combination of one or morememory devices, short term, and/or long term memory. Input system 112Cmay be any combination of input devices, such as one or more keyboards,mice, trackballs, scanners, cameras, and/or interfaces to networks.Output system 112D may be any combination of output devices, such as oneor more monitors, printers, and/or interfaces to networks. As shown byFIG. 2, system 116 may include a network interface 120 (of FIG. 1)implemented as a set of HTTP application servers 200, an applicationplatform 118, tenant data storage 122, and system data storage 124. Alsoshown is system process space 202, including individual tenant processspaces 204 and a tenant management process space 210. Each applicationserver 200 may be configured to tenant data storage 122 and the tenantdata 123 therein, and system data storage 124 and the system data 125therein to serve requests of user systems 112. The tenant data 123 mightbe divided into individual tenant storage areas 212, which can be eithera physical arrangement and/or a logical arrangement of data. Within eachtenant storage area 212, user storage 214 and application metadata 216might be similarly allocated for each user. For example, a copy of auser's most recently used (MRU) items might be stored to user storage214. Similarly, a copy of MRU items for an entire organization that is atenant might be stored to tenant storage area 212. A UI 230 provides auser interface and an API 232 provides an application programmerinterface to system 116 resident processes to users and/or developers atuser systems 112. The tenant data 123 and the system data 125 may bestored in various databases, such as one or more Oracle™ databases.

Application platform 118 includes an application setup mechanism 238that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage122 by save routines 236 for execution by subscribers as one or moretenant process spaces 204 managed by tenant management process 210 forexample. Invocations to such applications may be coded using PL/SOQL 234that provides a programming language style interface extension to API232. Invocations to applications may be detected by one or more systemprocesses, which manages retrieving application metadata 216 for thesubscriber making the invocation and executing the metadata as anapplication in a virtual machine.

Each application server 200 may be communicably coupled to databasesystems, e.g., having access to system data 125 and tenant data 123, viaa different network connection. For example, one application server 200₁ might be coupled via the network 114 (e.g., the Internet), anotherapplication server 200 _(N-1) might be coupled via a direct networklink, and another application server 200 _(N) might be coupled by yet adifferent network connection. Transfer Control Protocol and InternetProtocol (TCP/IP) are typical protocols for communicating betweenapplication servers 200 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network interconnect used.

In certain embodiments, each application server 200 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 200. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 200 and the user systems 112 to distribute requests to theapplication servers 200. In one embodiment, the load balancer uses aleast connections algorithm to route user requests to the applicationservers 200. Other examples of load balancing algorithms, such as roundrobin and observed response time, also can be used. For example, incertain embodiments, three consecutive requests from the same user couldhit three different application servers 200, and three requests fromdifferent users could hit the same application server 200. In thismanner, system 116 is multi-tenant, where system 116 handles storage of,and access to, different objects, data and applications across disparateusers and organizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 116 to manage their salesprocess. Thus, a user might maintain contact data, leads data, customerfollow-up data, performance data, goals and progress data, etc. allapplicable to that user's personal sales process (e.g., in tenant datastorage 122). In an example of a MTS arrangement, since all of the dataand the applications to access, view, modify, report, transmit,calculate, etc., can be maintained and accessed by a user system havingnothing more than network access, the user can manage his or her salesefforts and cycles from any of many different user systems. For example,if a salesperson is visiting a customer and the customer has Internetaccess in their lobby, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 116 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS should have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant specific data, system 116 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 112 (which may be client systems)communicate with application servers 200 to request and updatesystem-level and tenant-level data from system 116 that may requiresending one or more queries to tenant data storage 122 and/or systemdata storage 124. System 116 (e.g., an application server 200 in system116) automatically generates one or more SQL statements (e.g., one ormore SQL queries) that are designed to access the desired information.System data storage 124 may generate query plans to access the requesteddata from the database.

Example Testing Methods

Referring now to FIG. 3, shown is an example simplified flow diagram ofa method for testing a module, such as in a multi-tenant databaseenvironment. Source code of a test framework that is configured fortesting the module may be accessed as indicated at block 300. Aconfiguration folder having a property override for a test suite for themodule testing can be created as indicated at block 302, and a sourceroot folder for the test suite may be determined as shown in block 304.The test framework can be started by passing in an identifier for thetest suite, as indicated at block 306, and a custom test to the sourceroot folder may be added to customize the test suite using theconfiguration folder, as shown in block 308.

Referring now to FIG. 4, shown is a more detailed flow diagram of theexample shown in FIG. 3. The source code of the test framework may beaccessed, as indicated at block 300, and the configuration folder havingthe property override for the test suite may be created, as shown inblock 302. A check can be made at block 400 to verify whether apredetermined test suite is defined for a customized test suite. If thecheck made at block 400 is evaluated to “Yes,” the predetermined testsuite may be extended for the customized test suite, and a router orpath can be added to another source root folder, as indicated at block402. However, if the check made at block 400 is evaluated to “No,” asource root folder may be created for the customized test suite, asindicated at block 404. Finally, the source root folder for the testsuite may be determined, as shown in block 304. The test framework canbe started with each of a plurality of test folders enabled forauthoring changes to a library associated with the source code for thetest framework, as indicated at block 406. The test framework may becompiled with each of the plurality of test folders enabled, asindicated at block 408. Changes are made in a test framework file, e.g.,by utilizing a refactoring tool, as indicated at block 410, and thecustom test to the source root folder may be added to customize the testsuite using the configuration folder as shown in block 308.

The particular example method for testing a module that is described inFIGS. 3 and 4 can utilize a concept where all software built on top ofthe test framework is stored with the test framework in the same sourcecode repository. This can accommodate drastic code changes in the testframework to occur using modern refactoring tools, and those refactoringtools are able to properly modify the source code that was built on topof the test framework. This is made possible because that source code isstored in the same place as the test framework. This gives the authorsof the test framework a substantially larger degree of freedom in thetypes of changes that can be made. When a test suite is selected, onlythe source code that is relevant for that test suite may be compiled andincluded in the development environment. This shortens the time requiredto prepare a test suite for execution.

The selected test suite affects the configuration values at runtime, andit is possible for the test suite to set configuration values for thattest suite only, which can help compartmentalize test suites that arebuilt on top of the core test framework. This configuration system uses“strong typing,” which is a practice of representing configurationproperty values using data types that are directly relevant for eachconfiguration property. Examples include character strings, numericintegers, URLs, and floating point values. A team that uses a specifictest suite can thus develop their tests in isolation from other teams.This includes the inability for another team's changes to other testsuites to break the ability to compile and build one's own test suite.Meanwhile, the authors of the test framework may retain the ability tosee all test suites' source codes for the purposes of test frameworkrefactoring and development.

Particular embodiments provide for isolation of test suites in a mannerthat gives individual test suites a high degree of individual freedom,while retaining the ability for the test framework authors to compilethe framework with the entire set of suites selected. In this way, abalance between competing attributes of maximizing test suiteflexibility while maximizing the freedom of the test framework authorsto significantly refactor the test framework, may be achieved. Further,example methods combined the test suites and the test framework into acommon codebase, while adding test suite isolation features, therebyyielding a balance that achieves the best of both competing attributes.

Using configuration metadata and a separation of test suites, the testframework's set of strongly typed configuration values and availablesource code of the test framework described the FIGS. 3 and 4 can differbased upon the presently selected test suite. Example methods adjust thetest framework, provide for centralized modifications, and also permitmultiple teams to use the same testing framework without having to spendtime in compiling tests or seeing configuration keys that belong toother suites. Refactoring the test framework becomes easier with thisapproach as it is possible to select all suites and make sweeping codechanges with one check-in instead of requiring the framework-using teamsto make their own changes for the refactoring or worry about providingbackward compatibility.

Example Test Framework Adjustment and Configuration

Particular embodiments can be implemented in a “ProdTest” testframework, as exemplified below. The ProdTest's properties system allowfor configuration of ProdTest's behavior in multiple places usingproperties whose metadata are defined in a single property definitionfile. Those multiple places include default values in theproperty-definitions.xml file, *.properties files, test environmentproperty files, test instance property files, test suite property files,and the command line. With some exceptions, all properties can be set inany of the aforementioned places, though most properties tend to makemore sense in one or two places than in others. Java code withinProdTest, including tests, can retrieve property values using stronglytyped property accessors in the Properties class, which can be retrievedeasily via Globals.get( ). A test that needs to connect to anotherinstance can do so by using the information from a new Properties objectinstance constructed by the test by passing in the desired environment,instance, and suite to load the properties for. Tests that are meant torun only when specific conditions about an instance are true can performthat by using the InstanceRef enum with the @OnlyTheseInstances and/or@SkipInstances annotations. Any property can appear in InstanceRef whenits instanceRef attribute in property-definitions.xml is set to true.

This properties system may utilize code generation to convert theproperties definition file along with the sets of known environments andinstances into Java class files. For example, such target files caninclude:

-   -   (1) Properties: Property getters and setters;    -   (2) InstanceRef: Enum for properties that have instanceRef set        to true in property-definitions.xml, which can be used in        @OnlyTheseInstances and @SkipInstances.    -   (3) Environment: Enum for the environments that exist in the        config/environments folder;    -   (4) Instance: Enum for the instances that exist across all        environments;    -   (5) EnvironmentInstance: Enum for the known environment and        instance combinations;    -   (6) Region: Enum for the known regions, which are taken from the        alphabetic prefix of instance names that have an alphabetic        prefix followed by a numeric suffix. Not all instances have a        region.

The enums listed above include methods that allow navigating from one tothe other. Each has a relationship with the other, so those methods wereadded for the convenience of enumerating an enum's related enums.ProdTest integrations that use older property names in *.propertiesfiles and command line arguments continue to work as a result of limitedbackward compatibility.

When properties are loaded, different places override each other in aspecific order and are arranged in different blocks. Properties that areset in lower blocks override properties having the same names as thosethat were set in upper blocks. As an example, the command lineproperties trump the defaults from property-definitions.xml. Examples ofsuch different blocks can include:

-   -   (1) Construction arguments: These may be passed into the        Properties object at construction time, and these properties are        environment, instance, and suite. A no-argument constructor        overload gets the environment, instance, and suite properties        from the command line or, if not present on the command line it        can get from the *.properties files. Once set, the environment,        instance, and suite properties cannot be overridden. In        property-definitions.xml, they may have their overridable        attributes set to false.    -   (2) Defaults from property-definitions.xml: Property definitions        can optionally set a default value. These default values can        contain property references, which can be the same as the other        places where property values can be set.    -   (3) Basic properties: The process of loading the properties also        depends on the values of some key properties. These can include        configDir, environment, environmentInstance, environmentsDir,        instance, prodTestHome, region, suite, and suitesDir. The values        for the basic properties may have their defaults rooted in        property-definitions.xml, though they can be overridden within        the *.properties files. Basic properties, with the exception of        environment, instance, and suite, may not be overridden using        the command line. Other properties can be overridden using the        command line. Before the properties loader can read the        environment, instance, and suite properties, extended markup        language (XML) files know their base directory paths, and the        basic properties included in those paths. The basic properties        can have their property substitutions performed at this point so        that the final values are known before the subsequent steps in        the properties loading process are executed.    -   (4) Environment: If the environment being loaded has an        environment.xml file, then the properties within it may be read.        An environment does not need to have an environment.xml file. An        environment may have any number of nested subenvironments. Each        subenvironment similarly may have an optional environment.xml        file that applies to that subenvironment and any of its        subenvironments. Each subenvironment can set any property or        override properties from any ancestor environment, and those        properties can be overridden by any descendant subenvironment,        instance, suite, *.properties file, or command line.    -   (5) Instance: An instance that exists within an environment        folder may include its properties XML file loaded. Every        environment instance has a properties XML file, even if it        contains no property overrides. The existence of an instance        properties XML file within an environment folder can be similar        to an instance's existence configured in ProdTest.    -   (6) Suite: If the suite being loaded has a suite.xml file, then        the properties within it can be read. The suite typically has a        suite.xml file, but it is not required. The suite.xml file for        the suite typically contains property overrides that are        relevant to that suite, but not to specific environments or        instances.    -   (7) *.properties files: The property overrides that exist in        *.properties files, such as default.properties and        user.properties, can be used to persist property overrides        without having to modify the environment, instance, or suite        properties files. The override order for the *.properties files,        with the last one processed being last, e.g., being as shown        below:        -   (A) default.properties: The defaults for the basic            properties may live in this file along with some additional            information that is required to bootstrap Ant.        -   (B) user.properties: The user.properties stores persisted            customizations. This file can be writable by default so that            there is no need to check it out to edit it. In most cases,            there is no need to edit this file, but for setting            persistent defaults, such as the environment, instance, or            suite to use, then they can be set here. The command line            can be used to override those and other properties.        -   (C) build.properties: The ProdTest servers may be in the SFM            datacenter, which can be controlled using ProdTest's pages            on Luna, and put their own overrides into this file.        -   (D) server.properties: Various properties of the server.

In addition, the substitutions for these properties can be handled by an“ant” Perl script, and the resulting values maybe passed onto Ant andProdTest via environment variables. Further, another example propertyblock includes Command line: Properties that are not basic can beoverridden by using command line system property arguments. As someexceptions, environment, instance, and suite, which are basicproperties, can be overridden via the command line. These can take theform of: DpropertyName=value on the command line. Property names thatare passed in via command line arguments may be case insensitive. Thiscan be different from the other places where property overrides canexist, since the property names are case sensitive everywhere else.Legacy property names are permissible in the command line arguments. Inother places where property overrides can exist, legacy property namesare generally not permissible, though specific exceptions exist for*.properties files for backward compatibility. Behind the scenes, Antmay pass all system property command line arguments that it was givenalong to ProdTest.

Example properties defined in property-definitions.xml have getters andsetters generated for them in Properties.java, excepting properties thatare defined to have no getter or setter. The definitions inproperty-definitions.xml may also include the data type, and thegenerated getters and setters in Properties.java which accept and returnthose data types thereby making their use in ProdTest easier.Alternatively, it is possible to access property values throughgetProperties( ), getAllProperties( ), and substitute( ). One suchproperty is getters and setters: example code generated for xyz inProperties.java if an integer property named maxTestAttempts is definedin property-definitions.xml, and neither the getter or setter has beendisabled is given below:

/** * Gets the maximum number of attempts that will be made to run afailing test before marking the test as a failure. * This gives a testanother chance to pass and, if it does pass, the test will be marked aspassed. * <br/><br/> * The default value is 1 * <br/><br/> * Thisproperty can be written to with setMaxTestAttempts( ). * @return Thecurrent value of the maxTestAttempts property * @seetest.util.Properties#setMaxTestAttempts */ public intgetMaxTestAttempts( ) { return (Integer)properties.get(“maxTestAttempts”); } /** * Sets the maximum number ofattempts that will be made to run a failing test before marking the testas a failure. * This gives a test another chance to pass and, if it doespass, the test will be marked as passed. * @param value The new value toconfigure * @see test.util.Properties#getMaxTestAttempts */ public voidsetMaxTestAttempts(int value) { properties.put(“maxTestAttempts”,value); }

The example code above can permit code elsewhere in ProdTest to retrieveits value as a primitive integer without having to convert it. Whensetting values, it may be the caller's responsibility to ensure that thevalues being set are immutable and thread-safe. The object typessupported in the properties system can ensure this in most cases, butfor string lists, extra work may be performed on the caller's part.Instead of setting a string list property using any plain mutable list,such as ArrayList, please make an immutable copy withImmutableList.copyOf( ) and pass that copy into the property setter. Ifthis is not done, another thread that retrieves the string list latermay not have the most up-to-date values in the string list elements.

Another example property is the property map: The property map returnedby getProperties( ) can be a live view of the underlying map withvisibility and mutability restrictions applied. If a getter does notexist for a property, then it may not be present in this map. If asetter does not exist for a property that does have a getter, then itwill be present in this map, but it cannot have its value updated.Hidden properties are absent from this map, though getAllProperties( )contains hidden properties. To determine which properties are read-writeand which are read-only, Properties.java can include static getters thatreturn those property name sets. A properties user interface can usethat information for mutability decorations, among other things.

Another example property is Read-Only Property Map: The property mapreturned by getAllProperties( ) can return an unmodifiable view of theproperty map, which includes hidden properties. This map may not be asexpensive to work with as the map returned from getProperties( ), but itcannot be written to. To determine which properties are read-write andwhich are read-only, Properties.java includes static getters that returnthose property name sets. A properties user interface can use thatinformation for mutability decorations, among other things.

Another example property is Property Reference Substitution: Stringsthat contain property references within them can have those propertyreferences substituted with their currently configured values with anexample substitute( ) method. Such a method may be utilized for specialcases as it is far more efficient to call a property's getter than it isto call the substitute( ) method. Property references can use the samesyntax that they do in the property, *.properties, and users.xml files.As an example, calling substitute( ) with the “Hello, ${instance} on${environment}” string replaces ${instance} with the name of the currentinstance and ${environment} with the name of the current environment. Ifthe current configuration is NA1 on the qatest environment, then thestring returned from substitute( ) would be “Hello, NA1 on QATEST”. Notethat the upper-case forms of the environment and instance names can beused in this example because those are the enum value names.

The strings from the property files and command line arguments may nolonger be around by the time that substitute( ) can be called, sosubstitute( ) uses toString( ) on the strongly typed property value toconvert it into a string. As a result, it may be best to refer to stringor primitive properties in property references in substitute( ). Callingsubstitute( ) with a string list property reference, as an example, willsubstitute that property reference with the comma separated list ofelements enclosed in square brackets, which might not be what the callerwants.

As a way to determine where some test cases execute, InstanceRef enumvalues can be referred to in @OnlyTheseInstances and @SkipInstancesclass or method level annotations. The property-definitions.xml filedrives which enum values are generated in InstanceRef, and theproperties that can be put into InstanceRef are those that have theirinstanceRef attribute set to true. Different property data types may behandled in their own unique ways when put into InstanceRef, as discussedin the general data types below.

One example general data type is Enum: An InstanceRef enum value may becreated for every enum value in the property's enum data type. TheInstanceRef enum value name can be a concatenation of the property name,an underscore character, and an enum value in the property's enum. As anexample, if the “instance” property has instanceRef set to true, thenevery instance will have its own InstanceRef enum value. A more concreteexample is InstanceRef.INSTANCE_NA1, which is applicable when theinstance from Globals.get( ).getInstance( ) is Instance.NA1.

The Environment enum can be handled in a special way. An environmentInstanceRef is considered applicable if the current environment is thereferenced environment or any of its descendant subenvironments. Thismeans that InstanceRef.ENVIRONMENT_QATEST may be considered applicableif the current environment is either Environment.QATEST orEnvironment.QATEST_SANDBOX. If the referenced environment, however, isInstanceRef.ENVIRONMENT_QATEST_SANDBOX, then onlyEnvironment.QATEST_SANDBOX makes the instance applicable.Environment.QATEST may not be a descendant ofEnvironment.QATEST_SANDBOX.

Another example general data type is Object: If the property's value atruntime is not its blank equivalent, then the InstanceRef for it may beconsidered applicable. This can apply as well to strings, where a stringmay be non-blank for its InstanceRef to be considered applicable.Similarly, a non-empty list would make a stringlist applicable, and aproxy other than Proxy.NO_PROXY would make a proxy server propertyapplicable.

Another example general data type is Boolean: If the property's value atruntime is true, then the InstanceRef for it may be consideredapplicable. Another example general data type is Integer: If theproperty's value at runtime is not 0, then the InstanceRef for it can beconsidered applicable. Another example general data type is Double: Ifthe property's value at runtime is not 0.0, then the InstanceRef for itmay be considered applicable.

With the exception of enums, a property with instanceRef set to true cangenerate one InstanceRef enum value. As an example, if the “sandbox”property has instanceRef set to true, then a test that can only run on asandbox instance can use: @OnlyTheseInstances(InstanceRef.SANDBOX) torun on only sandbox instances, where the properties files set the“sandbox” property to true, e.g., on those instances that are sandboxinstances.

Example environments that ProdTest knows about may be in theconfig/environments folder as subfolders, and the name of a subfolder inthat folder concatenated with any recursive subfolders using anunderscore character as the delimiter can be the environment name. Thatinformation may be copied into the generated Environment.java class. Inthe Environment.java class, one can get the instances within aparticular environment through the getEnvironmentInstances( ) method,which returns a map of Instance objects to EnvironmentInstance objects.One can get the current environment through Globals.get().getEnvironment( ). Example instances that ProdTest knows about may becollected from the config/environments folders, aggregated, and writtenout to the generated Instance.java class file. An instance in this enummay not be a direct instance within an environment. Instead, it may bean abstraction that allows an instance that appears in multipleenvironments to be addressed as such. As an example, Instance.NA1 refersto all NA1 instances, regardless of the environment.

The Instance.java class may also have a getter for the region, which canreturn the Region enum value for this instance if one exists. Not allinstances may have a region. For example, those instances that areprefixed with alphabetic characters and suffixed with numbers areconsidered to have a region, and the region is the alphabetic prefix.One can get the current instance through Globals.get( ).getInstance( ).An environment instance is a specific instance within an environment.Another way to look at an environment instance is as a junction betweenthe many-to-many relationship between environments and instances. Eachenvironment instance enum value has one environment and one instance,and both are retrievable with getters.

As an example method, the static getEnvironmentInstance( ) methodreturns the EnvironmentInstance enum value for the input Environment andInstance enum values. Various methods throughout ProdTest require anEnvironmentInstance as a parameter since EnvironmentInstance is aconvenient enum which refers to both an environment and an instance. Onecan get the current environment instance through Globals.get().getEnvironmentInstance( ). ProdTest can consider instances that follownaming conventions to have no region which is defined as the alphabeticprefix of an instance that includes an alphabetic prefix with a numericsuffix. In the Region.java class, one may get the instances that aregion contains via the getInstances( ) method. Because regions arederived from the instances that ProdTest finds in theconfig/environments subfolders, every region has at least one instance.One may also get the current region through Globals.get( ).getRegion( ).

A suite in ProdTest may be a general use of the ProdTest framework.These include, but are not limited to, production testing, upgradetesting, and testing of the ProdTest framework. Suites can exist assubfolders in the config/suites folder, and the name of the subfolder isthe suite's name. ProdTest may load a suite's configuration by readingthe suite.xml file within a test suite folder. Example files that existin a typical suite subfolder within config/suites are the following:

-   -   (1) suite.xml: A properties file that is processed after loading        the environment and instance properties. Properties are        indicated that can be relevant to a suite, regardless of the        environment or instance. This file may define a sourceSuffixes        property value, which references the source folder suffixes to        java/src that should be included in the build and runtime        classpaths whenever this suite is selected. An example is        “prodtest,prodtest-test”, which would effectively add java/src,        java/src-prodtest, and java/src-prodtest-test to the classpath        as source folder roots, and the classpath would get        java/classes, java/classes-prodtest, and        java/classes-prodtest-test added.    -   (2) test-inventory.xml: This may be the default test inventory        file for the suite. This can be overridden with the        “testInventory” property.    -   (3) users.xml: This can be the user list file for the suite, and        can be overridden with the “users” property.

The above files excepting suite.xml, can exist elsewhere if suite.xmltells ProdTest to look elsewhere for them, and their filenames can bedifferent, too. Generally, each suite may have its own source folderroot in java/src-suffix, where “suffix” is the name of the test suite.It does not need to be the name of the test suite, but it might help tokeep things consistent. The suite.xml file for the suite can define thesource folder suffixes to include in the build and class paths, and thisproperty is named sourceSuffixes. If suite “xyz” has its source codestored in java/src-xyz, then sourceSuffixes should be set to xyz in thatsuite's suite.xml file.

The property file format may be used in the environment, instance, andsuite property XML files. This represents slightly less verbose than thejava.util.Properties XML file format, though it may be similarly akey-value pair list expressed as XML. Element names can be propertynames, and the inner content of those elements are their values. Anexample is the following:

<?xml version=“1.0” encoding=“UTF-8”?> <properties><ccList>prodtestcclist, mbenioff</ccList> <password>test{circumflex over( )}1234z</password> </properties>

The above example sets ccList to “prodtestcclist, mbenioff” and passwordto “test^1234z”. Nested properties may also be supported in specificcases. As an XML file, values can use XML escape sequences to avoid theinadvertent introduction of markup. As an example, to set the passwordproperty to a value of </password><hello type=“wo&rld”>, it will need tobe escaped at a minimum to &lt;/password>&lt;hello type=“wo&amp;rld>. Inthis example, < was escaped to &lt; and & was escaped to &amp;. To beconsistent, the > and ” characters can also be escaped.

Property values may be expressed as strings before being converted intostrongly typed objects at the end of the property loading process. Asstrings, special considerations must be taken into account. Example mainproperty values used are as shown below.

One example main property value is Escaping: Property values usebackslash escaping. Generally, when a backslash is encountered, the nextcharacter is used as-is in the property's value. As exceptions to thatrule, the characters n, r, and t may replace the escape sequence with anewline, carriage return, or a tab, respectively. As an example, if aproperty needed to set its property to the following text “hello\worldhow are you? Fine, I hope,” then the property value can be set using thefollowing value, which can unescape to the value above“hello\\world\nhow are you?\tFine, I hope.” The substitution reference\${configDir} can give us the configDir property value, and \!{fdqn}gives us the fully qualified domain name. Escaping includes propertyvalues Stringlist and XML properties, and are explained below.

Stringlist properties can add another layer of unescaping using almostthe same strategy as regular property values. This is because the entirevalue is unescaped before the stringlist code performs its ownunescaping. A key difference in the two layers' unescaping strategiesmay involve the stringlist's optional use of enclosing a string elementin double-quotes to include commas without requiring that they should beescaped with a backslash. As a result of this, stringlist elements canappear to be double-escaped in a stringlist property's overall valuestring.

As an example, if there is a stringlist with the first element set to“hello\world, how are you?” and the second element set to“hello[newline]world”, where [newline] is a new line, then the propertyvalue for the stringlist would be the following: hello\\\\world\\, howare you?,hello\\nworld. The first level of unescaping happens on theentire value, converting it into hello\\world\, how areyou?,hello\nworld. The second level of unescaping happens as elementsare read from the list. As a result, the “\,” subsequence is unescapedto a single comma in the first element. If it was not escaped, then thestringlist might have thought that it had three elements instead of two.

XML properties: When a property value is set in a property XML file, XMLescaping can also apply. The main characters that utilize escaping in aproperty XML file are the less than < and ampersand & characters, whichget escaped to &lt; and &amp;, respectively. Other characters may notneed to be escaped with XML escape sequences.

Another example main property value is property values that can refer toother properties, and those properties can refer to other properties.Circular references may not be supported, and if any are detected, anIllegalArgumentException is thrown during the property loading. To referto another property, enclose the property name with a ${ prefix and a }suffix. As an example, to include the resultsDir property in thedvtResultsDir property, the dvtResultsDir property can have its valueset in a property file with the following XML:<dvtResultsDir>${resultsDir}/DVT</dvtResultsDir>. Note that the propertynames may be case sensitive. If an unknown property name is referencedin a substitution, then an IllegalArgumentException will get thrownduring the property loading. Any place in the property loading processwhere property values are set can use substitutions, including specialsubstitutions. This includes command line arguments. As one exception,however, property substitutions that exist in a *.properties file mayrefer to another property in a *.properties.file, which is a result ofthe “ant” Perl script performing substitutions on those propertiesbefore Ant is run, which can be early in ProdTest's execution.

Another example main property value is special substitutions: examplespecial substitutions available in ProdTest are the following:

-   -   (1) Host: The name of the current host without the domain        suffix. As an example, if the system's hostname is        mbenioff-ws.internal.salesforce.com, then this special        substitution's value is mbenioff-ws. This substitution can be        the same or similar to the one in SFDC's app server settings        system.    -   (2) fqdn: This is the name of the current host with the domain        suffix. This special substitution's name is an acronym for the        host's fully qualified domain name (FQDN). As an example, if the        system's hostname is mbenioff-ws.internal.salesforce.com, then        this special substitution's value is        mbenioff-ws.internal.salesforce.com.    -   (3) appVersionFromServer: This is the API version of the server        behind the “hostname” property's value. If the API version could        not be retrieved, then a warning may be logged, and the API        version from the partner.wsdl file is returned instead.    -   (4) appVersionFromPartnerWsdl: This is the API version from the        partner.wsdl file. The code behind this reads the value from        com.sforce.soap.partnerConnector.END_POINT, which can be        generated from the partner.wsdl file during ProdTest's        compilation, so this may be more of an indirect reference. That        is, the partner.wsdl file may not actually be read as a result        of using this special substitution.

The particular method described in FIGS. 3 and 4 can utilizeproperty-definitions.xml file stored in the config folder, plays acentral role in the PropertiesCodeGenerator's processing. Within itsroot properties element, each property element defines a singleproperty. The attributes on a property element set the property'smetadata. The optional inner text of a property element is thatproperty's documentation, which gets used in the JavaDocs for itsgetters and setters in the Properties.java class. An exampleproperties-definition.xml file is the following:

<?xml version=“1.0” encoding=“utf-8” ?> <properties> <propertyname=“environment” type=“environment” setter=“false” instanceRef=“true”overridable=“false” basic=“true”> The environment being tested.</property> <property name=“entityTestConfigDir”default=“${configDir}/entitytest” type=“file”> The configuration folderfor the entity tests. </property> <property name=“uploadToQaForce”default=“false” type=“boolean”> Whether or not the test cases that arerun will be uploaded to QaForce upon the test run's completion.</property> <property name=“seleniumBTPassword” type=“string”/></properties>

The above illustrative example, which may miss a number of keyproperties, uses several data types and combinations of metadataconfigurations, such as: environment, entityTestConfigDir, java.io.Filedata type, uploadToQaForce, and seleniumBTPassword.

Example attributes that are possible on a property attribute include thefollowing. One example attribute is name: name refers to name of theproperty, which maybe be a camelCase alphanumeric name that starts witha letter.

Another example attribute is Type: Type refers to the data type of theproperty. This attribute value is case insensitive. The blank values mayalso include values having entirely of whitespace characters. Suchexample types used are: appversion, Boolean, double, environment,Environmentinstance, executionmode, file, host, instance, integer,proxy, region, seleniumconfiglist, string, stringlist, url, users, andadding a new data type.

For the example type appversion: It comes under Java type:common.api.AppVersion and its property string value may be a PI versionnumber, such as 11.1 or 16.0. Its value when string is blank maybe null.

For the example type Boolean: It comes under Java type: Boolean and itsproperty string value is true or false. Its value when string is blankcan be false.

For the example type double: It comes under Java type: double and itsproperty string values are real numbers expressed as a string, such as42 or 3.14159. Its value when string is blank may be 0.0.

For the example type environment: It comes under Java type:test.util.Environment and its property string values are any enum valuethat exists in the Environment enum, such as qatest. Its value whenstring is blank can be null.

For the example type Environmentinstance: It comes under Java type:test.util.EnvironmentInstance and its property string values are anyenum value that exists in the EnvironmentInstance enum, such asproduction_na1. Its value when string is blank may be null.

For the example type executionmode: It comes under Java type:test.util.ExecutionMode and its property string values are any enumvalue that exists in the ExecutionMode enum, which are ftest, selenium,and all. Its value when string is blank can be null.

For the example type file: It comes under Java type: java.io.File andits property string values are Path to a directory or file, such as${prodTestHome}/config or ${dotDir}/generic.dot.zip. Its value whenstring is blank may be null.

For the example type host: It comes under Java type: java.lang.Stringand its property string values are any hostname or IP address. A portnumber followed by a colon is optional. Unsupported characters forhostnames are converted into hyphens. If the string is not in thehostname or hostname:port format, then an IllegalArgumentException maybe thrown. Its value when string is blank can be null.

For the example type instance: It comes under Java type:test.util.Instance and its property string values can be any enum valuethat exists in the Instance enum, such as na1 or cs3. Its value whenstring is blank may be null.

For the example type integer: It comes under Java type: int and itsproperty string values can be any integer number expressed as a string,such as 3 or 90. Its value when string is blank may be 0.

For the example type proxy: It comes under Java type: java.net.Proxy andits property string values can be proxy server and port expressed ashostname:port, such as myproxy.mydomain.com:8080. Its value when stringis blank may be null.

For the example type region: It comes under Java type: test.util.Regionand its property string values can be any enum value that exists in theRegion enum, such as na or ap. Its value when string is blank may benull.

For the example type seleniumconfiglist: It comes under Java type:java.util.List<test.util.SeleniumConfig> and its property string valuescan be, e.g., selenium configuration comma separated list with eachelement expressed as hostname:port:maxConcurrentSessions:browserStart,where hostname:port addresses a selenium server, maxConcurrentSessionsis the maximum number of concurrent sessions to send that seleniumserver, and browserStart is the browser start string to send to theselenium server. As an example, my-selenium-server:4444:5:*firefoxproxywill send up to five concurrent selenium sessions to my-selenium-serveron port 4444, and the browser that will be used is Mozilla Firefox. Asanother example,“my-selenium-server:4444:5:*firefoxproxy,another-selenium-server:4444:10:*firefoxproxy/usr/local/firefox3/firefox” (without the double quotes) will also sendup to 10 selenium sessions to another-selenium-server:4444, and MozillaFirefox will be started by explicitly invoking/usr/local/firefox3/firefox instead of having selenium find whereFirefox exists. With the second selenium server included, ProdTest canalso load balance sessions between those two selenium servers, and morethan two selenium servers can be used for a browser. Its value whenstring is blank may be null.

For the example type string: It comes under Java type: java.lang.Stringand its property string values can be any string value. Note that in aproperties XML file, the < and & characters must be escaped properlywith &lt; and &amp; respectively. Its value when string is blank may benull.

For the example type stringlist: It comes under Java type:java.util.List<java.lang.String> and its property string values can becomma separated list of strings with backslash escaping. When abackslash is encountered, the next character is copied as-is unless ifit is n, r, or t, which are converted to a newline, a carriage return,or a tab, respectively. If a double-quote character is encountered, thencharacters up to the next double-quote character are part of theelement, including any commas. Its value when string is blank may benull.

For the example type url: It comes under Java type: java.lang.String andits property string values are comma separated list of strings withbackslash escaping. A string can be formatted as a URL, such ashttp://na1.salesforce.com. Characters within the URL's host part thatare not valid hostname characters will be converted into hyphens. If thestring is not formatted as a recognizable URL, then anIllegalArgumentException is thrown. The java.net.URL class was not usedbecause it is mutable and is rather inefficient for how ProdTestprimarily uses URLs, which is with string concatenation. Its value whenstring is blank may be null.

For the example type users: It comes under Java type: test.util.Usersand its property string values may be Path to a users.xml file, such as${suitesDir}/${suite}/users.xml. Its value when string is blank can benull.

For the example type of adding a new data type: To add a new data type,a new enum value is added tobuild.PropertiesCodeGenerator.PropertyDataType and totest.util.PropertyDataType. The name of the new enum value may be equalin both enum classes. The former can be used during code generation tostore information about the Java type information for the property aswell as how to cast the return value in the generated getter, and thelatter may be used at runtime to convert a property's string value intoa strongly typed Java object. In general, a property value that isnonexistent, blank, or having entirely of whitespace must map to null.If the property value string does not properly describe a constructibleinstance of the data type, then an IllegalArgumentException may bethrown.

Another example attribute is getter: When the value of getter is true,which is the default, a getter will be generated for this property inProperties.java. The map returned by the getProperties( ) method canalso include this property. When the values of getter is false, a getterwill not be generated, and the map returned by the getProperties( )method will not include this property. Such properties are known ashidden properties. When the getter attribute is false, the setterattribute may also be set to false. Otherwise, the code generator mightdisplay a warning and skip the code generation for such properties.Hidden property values can be accessed via the getAllProperties( )method in the generated Properties.java class, and the hidden propertynames can be accessed via the getHiddenPropertyNames( ) static method.

Another example attribute is setter: When the value of setter is true,which is the default, a setter may be generated for this property inProperties.java. When false, a setter may not be generated, and the mapreturned by the getProperties( ) method prevents updates to suchproperties. These properties are known as read-only properties. When thegetter attribute is false, the setter attribute must also be false.Write-only properties are not supported. The read-only property namescan be accessed via the getReadOnlyPropertyNames( ) static method, andthe read-write property names can be accessed via thegetReadWritePropertyNames( ) static method.

Another example attribute is default: The default value for theproperty, which can be used if nothing else overrides it. Foroverridable properties, this default value can be overridden by anyother place where a property can be set, such as in a properties XMLfile for an environment, instance, or suite, in a *.properties file, orvia the command line. For properties that are not overridable, setting adefault value in the property definition ensures that no other place canset the property. If this attribute is not present, then the defaultvalue will be null for object types, false for booleans, 0 for integers,and 0.0 for doubles.

Another example attribute is instanceRef: When the instanceRef: is true,this property can be included in the generated InstanceRef enum, whichcan be used in @ Only. TheseInstances and @SkipInstances annotations.This attribute may be false by default.

Another example attribute is basic: When the value of basic is true,this property may have its property substitution references resolvedearly in the property loading process. Only a very small set ofproperties may be meant to be basic, and these are the properties thatinfluence important dependencies in the property loading process, suchas where the config, environments, and suites folders are located. It isalmost safe to say that no more basic properties should be added to theproperty-definitions.xml file. If a basic property is to be added, thenthe approach is probably incorrect. This attribute may be set to falseby default.

Another example attribute is legacyNames: Comma delimited list of caseinsensitive property names that can be used on the command line as analternative to the property's defined name. This is primarily meant forbackward compatibility. As an example, the legacyNames attribute valuefor the testFilter property is ftest.runSingleTest. When ProdTest seesthe ftest.runSingleTest property on the command line, it will set thetestFilter property. This list uses backslash escaping. When a backslashis encountered, the next character is copied as-is unless if it is n, r,or t, which are converted to a newline, a carriage return, or a tab,respectively. If a double-quote character is encountered, then allcharacters up to the next double-quote character are part of the currentlegacy name, including any commas. Note that this only works withcommand line property arguments. Property files and *.properties filesmust use the exact case-sensitive property name as defined inproperty-definitions.xml, and legacy names will not be honored inproperty files and *.properties files. This attribute is null bydefault, which makes the property's name the only way to override suchproperties from the command line. The names of properties with nolegacyNames attribute are also case insensitive on only the commandline.

Another example attribute is overridable: When true, which is thedefault, this property can be overridden after being set. Setting thisto false may prevent the setting of such a property after the firstplace that sets a value for it. As an example, if a non-overridableproperty has its first value set in a test instance properties file,then an exception may be thrown if any subsequent place in theproperties loading order tries to set a value for that property. If adefault value is set in the property-definitions.xml file, then no otherplace can effectively override its value. This property attribute issimilar to the inverse of Java's “final” modifier.

Another example attribute is spaceDelimited: When true and when theproperty type is stringlist, each whitespace subsequence of a propertyvalue string can be converted into a single comma before the propertyvalue string gets deserialized into a List<String> collection. This isuseful for legacy properties that were primarily space delimited, andthis approach for supporting such properties effectively makes themdelimited by either spaces or comma. No escape sequence exists forspaces in the processing of the values of properties that have thisenabled, so use of this property attribute is not recommended for newproperties. If this is true on a property whose type that is not astringlist, then a warning can be displayed, and the property will beignored. This attribute can be set to false by default.

Usernames for different environment, instance, org, and user typecombinations can be accessed through the Users.java class by retrievingit via Globals.get( ).getUsers( ). The Users.java file may not begenerated, though it does read the users.xml file that the “users”property tells it to read.

The users.xml file allows any hierarchical combination of environment,instance, org, and username elements to set the usernames. Usernamesvalues can include substitution references, though the permissiblereferences are limited to environment, instance, and org, and theircontext is limited to the users.xml file that they appear in. That meansthat the substitutions in users.xml do not refer to properties. As anexample, if a test suite knows that the main username for any org isprodtest@${instance}.${org}.sfdctest, where ${instance} is any instancename on any environment and ${org} is the name of the org, then thisdefault username can be set without any environment or instance, such asin the following example:

<?xml version=“1.0” encoding=“utf-8” ?> <users> <username type=“main”value=“prodtest@${instance}.${org}.sfdctest” /> ...

Without having to specify any environments or instances, Users.javaknows the username for any org on any environment instance, though theorg names still must be defined in users.xml. We can make the aboveexample complete by adding the definition of the generic org. Some testsrely upon a default org name to be set, and the value of the“defaultOrg” property must equal the name of one org defined in the testsuite's users.xml. Note that org names are case insensitive.

<?xml version=“1.0” encoding=“utf-8” ?> <users> <username type=“main”value=“prodtest@${instance}.${org}.sfdctest” /> <org name=“generic” /></users>

While the above example works for many instances, some instances may nothave the instance name as a part of the username. NA0 is one suchexample, where the generic org's username might beprodtest@generic.sfdctest. To override the username for all NA0instances, regardless of the environment, the following can be used:

<?xml version=“1.0” encoding=“utf-8” ?> <users> <username type=“main”value=“prodtest@${instance}.${org}.sfdctest” /> <instance name=“na0”><username type=“main” value=“prodtest@${org}.sfdctest” /> </instance><org name=“generic” /> </users>

To add to this example, one might want to say thatprodtest@${org}.sfdctest also applies to development environments, suchas a localhost SFDC instance. The following example override for thedevelopment environment accomplishes this:

<?xml version=“1.0” encoding=“utf-8” ?> <users> <username type=“main”value=“prodtest@${instance}.${org}.sfdctest” /> <instance name=“na0”><username type=“main” value=“prodtest@${org}.sfdctest” /> </instance><environment name=“development”> <username type=“main”value=“prodtest@${org}.sfdctest” /> </environment> <org name=“generic”/> </users>

Sometimes, a special case breaks the default pattern. Using any desirednesting of elements, it is possible to add that override to users.xml,such as in this particular example:

<?xml version=“1.0” encoding=“utf-8” ?> <users> <username type=“main”value=“prodtest@${instance}.${org}.sfdctest” /> <instance name=“na1”><username type=“main” value=“prodtest@${org}.sfdctest” /><environment=“qatest”> <org name=“generic”> <username type=“main”value=“prodtest2@${instance}.${org}.sfdctest” /> </org> </environment></instance> <environment name=“development”> <username type=“main”value=“prodtest@${org}.sfdctest” / </environment> <org name=“generic” /></users>

The ${instance} and ${org} substitutions in prodtest2@${instance}.${org}is prodtest2@na1.generic.sfdctest, since it also can apply to any othercombination of org, environment, and instance. Since the nesting ordermay not matter when specifying all three parent elements for username,the element nesting could have been environment→org→instance,org→instance→environment, or others. When getUsername( ) on a Users.javainstance is called, such as with Globals.get( ).getUsers( ).getUsername(), the method determines the username based on the following order ofthe information known for a particular username element from theusers.xml file. The first match can be used, and the order prefers morespecific data over less specific data. At the end, the org, environment,and instance combination is cached so that subsequent lookups becomefast.

Substitutions may be performed at substantially the moment when org,environment, and instance are all known. This can occur during theprocessing of users.xml when usernames that have all three known areseen as well as during the getUsername( ) method call when missing datais filled in from the method's parameter arguments. If the users.xmlfile has multiple occurrences of the same amount of information for thesame org, environment, and instance names, then the last one thatappears in the file may be used. Attributes from the previous elementoccurrences, however, will be used if they are not overridden later.This means that if an org element for the generic org defines its dotfile, then a subsequently occurring org element also for the generic orgwill keep the previously set dot file attribute unless if it overridesit with a new value.

The users file format can include elements, such as org, environment,instance and username. In an example org element: The org element mayrepresent a ProdTest org that typically exists on every instance inevery environment. Examples include the generic ProdTest org and the B2IProdTest org. The edition, dot, and email values can includesubstitutions from any ancestor element, though attempts to reference avalue that does not exist in an ancestor element will result in anIllegalArgumentException. Example attributes of the org elements caninclude: name: case-insensitive name of the org; edition: org's edition,such as EE or DE; dot: relative name of the DOT file; and email: emailaddress to use for the prodtest user during the org signup process inProdTest.

In an example environment element: The environment element can the scopeto the environment named in the name attribute. The name attributespecifies the name of the environment, which may match an enum value inthe generated Environment.java class. This value can be caseinsensitive.

In an example instance element: The instance element can restrict thescope to the instance named in the name attribute. The name attribute ofthe instance can match an enum value in the generated Instance.javaclass. This value can be case insensitive.

In an example username element: The username element sets the usernamefrom the value attribute for the user type referenced in the typeattribute. The username value can include the environment, instance, andorg substitution references, and this is regardless of the usernameelement's ancestors, if any. Also, a password attribute may be added.Example attributes of the element Username include: value: specifies theusername for this username element, and can include substitutionreferences regardless of context; and type.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects according to the present invention. It should be understood that“table” and “object” may be used interchangeably herein. Each tablegenerally contains one or more data categories logically arranged ascolumns or fields in a viewable schema. Each row or record of a tablecontains an instance of data for each category defined by the fields.For example, a CRM database may include a table that describes acustomer with fields for basic contact information such as name,address, phone number, fax number, etc. Another table might describe apurchase order, including fields for information such as customer,product, sale price, date, etc. In some multi-tenant database systems,standard entity tables might be provided for use by all tenants. For CRMdatabase applications, such standard entities might include tables forAccount, Contact, Lead, and Opportunity data, each containingpre-defined fields. It should be understood that the word “entity” mayalso be used interchangeably herein with “object” and “table”.

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. In certain embodiments, forexample, all custom entity data rows are stored in a single multi-tenantphysical table, which may contain multiple logical tables perorganization. It is transparent to customers that their multiple“tables” are in fact stored in one large table or that their data may bestored in the same table as the data of other customers.

Any suitable programming language can be used to implement the routinesof particular embodiments including C, C++, Java, assembly language,etc. Different programming techniques can be employed such as proceduralor object oriented. The routines can execute on a single processingdevice or multiple processors. Although the steps, operations, orcomputations may be presented in a specific order, this order may bechanged in different particular embodiments. In some particularembodiments, multiple steps shown as sequential in this specificationcan be performed at the same time.

Particular embodiments may be implemented in a computer-readable storagemedium for use by or in connection with the instruction executionsystem, apparatus, system, or device. Particular embodiments can beimplemented in the form of control logic in software or hardware or acombination of both. The control logic, when executed by one or moreprocessors, may be operable to perform that which is described inparticular embodiments.

A “processor” includes any suitable hardware and/or software system,mechanism or component that processes data, signals or otherinformation. A processor can include a system with a general-purposecentral processing unit, multiple processing units, dedicated circuitryfor achieving functionality, or other systems. Processing need not belimited to a geographic location, or have temporal limitations. Forexample, a processor can perform its functions in “real time,”“offline,” in a “batch mode,” etc. Portions of processing can beperformed at different times and at different locations, by different(or the same) processing systems. A computer may be any processor incommunication with a memory. The memory may be any suitableprocessor-readable storage medium, such as random-access memory (RAM),read-only memory (ROM), magnetic or optical disk, or other tangiblemedia suitable for storing instructions for execution by the processor.

Particular embodiments may be implemented by using a programmed generalpurpose digital computer, by using application specific integratedcircuits, programmable logic devices, field programmable gate arrays,optical, chemical, biological, quantum or nanoengineered systems,components and mechanisms may be used. In general, the functions ofparticular embodiments can be achieved by any means as is known in theart. Distributed, networked systems, components, and/or circuits can beused. Communication, or transfer, of data may be wired, wireless, or byany other means.

It will also be appreciated that one or more of the elements depicted inthe drawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application. It isalso within the spirit and scope to implement a program or code that canbe stored in a machine-readable medium to permit a computer to performany of the methods described above.

As used in the description herein and throughout the claims that follow,“a”, “an”, and “the” includes plural references unless the contextclearly dictates otherwise. Also, as used in the description herein andthroughout the claims that follow, the meaning of “in” includes “in” and“on” unless the context clearly dictates otherwise.

While the invention has been described by way of example and in terms ofthe specific embodiments, it is to be understood that the invention isnot limited to the disclosed embodiments. To the contrary, it isintended to cover various modifications and similar arrangements aswould be apparent to those skilled in the art. Therefore, the scope ofthe appended claims should be accorded the broadest interpretation so asto encompass all such modifications and similar arrangements.

What is claimed is:
 1. A method for testing a module, the methodcomprising: accessing source code of a test framework that is configuredfor testing the module; creating a configuration folder having aproperty override for a test suite for the module testing; determining asource root folder for the test suite; starting the test framework bypassing in an identifier for the test suite; and adding a custom test tothe source root folder using the configuration folder to customize thetest suite.
 2. The method of claim 1, wherein determining the sourceroot folder comprises creating the source root folder if a predeterminedtest suite is not to be used for the customized test suite.
 3. Themethod of claim 2, wherein determining the source root folder comprisesadding a router to another source root folder if the predetermined testsuite is to be extended for the customized test suite.
 4. The method ofclaim 1, further comprising: starting the test framework with each of aplurality of test folders enabled for authoring changes to a libraryassociated with the source code for the test framework; and compilingthe test framework with each of the plurality of test folders enabled.5. The method of claim 4, further comprising: using a refactoring toolto make changes in a file within the test framework.
 6. A non-transitorymachine-readable storage medium having one or more instructions thereonfor testing a module, the instructions when executed by one or moreprocessors causing the one or more processors to carry out: creating aconfiguration folder having a property override for a test suite for themodule testing; determining a source root folder for the test suite;starting the test framework by passing in an identifier for the testsuite; and adding a custom test to the source root folder using theconfiguration folder to customize the test suite.
 7. The non-transitorymachine-readable storage medium of claim 6, wherein determining thesource root folder comprises creating the source root folder if apredetermined test suite is not to be used for the customized testsuite.
 8. The non-transitory machine-readable storage medium of claim 7,wherein determining the source root folder comprises adding a router toanother source root folder if the predetermined test suite is to beextended for the customized test suite.
 9. The non-transitorymachine-readable storage medium of claim 6, further comprising: startingthe test framework with each of a plurality of test folders enabled forauthoring changes to a library associated with the source code for thetest framework; and compiling the test framework with each of theplurality of test folders enabled.
 10. The non-transitorymachine-readable storage medium of claim 9, further comprising: using arefactoring tool to make changes in a file within the test framework.11. An apparatus for testing a module, the apparatus comprising: aprocessor; and one or more stored sequences of instructions which, whenexecuted by the processor, cause the processor to carry out: creating aconfiguration folder having a property override for a test suite for themodule testing; determining a source root folder for the test suite;starting the test framework by passing in an identifier for the testsuite; and adding a custom test to the source root folder using theconfiguration folder to customize the test suite.
 12. The apparatus ofclaim 11, wherein determining the source root folder comprises creatingthe source root folder if a predetermined test suite is not to be usedfor the customized test suite.
 13. The apparatus of claim 12, whereindetermining the source root folder comprises adding a router to anothersource root folder if the predetermined test suite is to be extended forthe customized test suite.
 14. The apparatus of claim 11, furthercomprising: starting the test framework with each of a plurality of testfolders enabled for authoring changes to a library associated with thesource code for the test framework; and compiling the test frameworkwith each of the plurality of test folders enabled.
 15. The apparatus ofclaim 14, further comprising: using a refactoring tool to make changesin a file within the test framework.